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ABSTRACT 



A method of adapting multimedia content to a client device, 
wherein the multimedia content includes one or more items 
and the client device has capabilities and resources associ- 
ated therewith, is provided. The method includes transcod- 
ing the multimedia content into a plurahty of transcoded 
content versions, wherein the plurality of transcoded content 
versions have different modalities and resolutions associated 
therewith. Next, the transcoded content versions that are not 
compatible with client device capabilities are fiUered out. 
Then, at least a portion of the resources associated with the 
client device are allocated among the one or more items of 
the multimedia content. Lastly, one or more of the 
transcoded versions of the multimedia content are selected 
to generate a customized content based on allocation of the 
cUcnt device resources. 

31 Claims, 5 Drawing Sheets 
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METHODS AND APPARATUS FOR 
ADAPTING MULTIMEDIA CONTENT FOR 
CLIENT DEVICES 

CROSS REFERENCE TO RELATED 
APPLICAHONS 

The present application is related to U.S. patent applica- 
tion Ser No. 09/413^15, entitled: "Method and Apparatus 
For Transcoding Multimedia Using Content Analysis," filed 
in the names of Li et al, on Oct. 6, 1999, which claims 
priority to U,S. provisional patent application Ser. No. 
60/103;303, entitled: "Method and Apparatus For Transcod- 
ing Multimedia Using Content Analysis," filed in the names 
of Li et al. on Oct. 6, 1998. 

HELD OF THE INVENTION 

The present invention relates to delivery of multimedia 
content to chent devices and, more particularly, to methods 
and apparatus for adapting such multimedia content for 
diverse client devices. 

BACKGROUND OF THE INVENTION 

Web documents delivered on the Internet are multimedia 
presentations that may include video, images, graphics, text 
and audio. Due to the recent rapid growth of devices that are 
connected to the Internet, there is a growing demand for 
providing universal access to such multimedia content to a 
wide variety of devices over a wide range of network 
environments. For example, personal computers on a local 
area network (LAN), personal digital assistants (PDAs) on 
dial-up modems and smart ceUular phones have drastically 
different client resources in terms of, for example, screen 
size, resolution, color depth, network bandwidth and com- 
puting power. Internet users also vary in their ability to pay 
for Internet services and in the time they are ready to wait 
for a page to download. Therefore, to provide universal 
access to the Internet, multimedia delivery methods need to 
accoimt for the composite nature of Web documents, and the 
variety of cUent platform capabiUties, user interests, network 
constraints and authoring policies. 

In this context, video-conferencing systems have been 
proposed that adjust the bandwidth available to the client by 
selecting a suitable compression factor or codec. In these 
systems, only a single type of multimedia item (namely 
video) is considered. Also, clients that can not handle video 
arc not considered. 

One option for content adaptation is to manually develop 
multiple versions of multimedia content, each suitable for a 
class of client devices. Given the variety of client devices, it 
is difiHcult for content publishers to anticipate and accom- 
modate the wide spectrum of client capabilities. For com- 
posite multimedia documents, such as Web pages, a number 
of systems have been proposed that employ a proxy between 
the Web server and the client. For example, various proxy 
approaches are described in: J. R. Smith, R. Mohan, and 
C-S. Li, "Transcoding Internet Content for Heterogeneous 
Chent Devices," In Proc. IEEE Inter. Symp. on Circuits and 
Syst. (IS CAS), Special Session on Next Generation Internet, 
June 1998; A. Fox, S. D, Gribble, E. A. Brewer, and E. Amir, 
"Adapting to Network and Client Variability Via 
On-demand Dynamic Distillation," In ASPLOS-VII, 
Cambridge, Mass., October 1996; A. Ortega, F. Carignano, 
S. Ayer, and M. VetterU, "Soft. Caching: Web Cache Man- 
agement Techniques for Images," In IEEE Workshop on 
Multimedia Signal Processing, pg. 475-480, Princeton, NJ., 
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June 1997; Intel Quick Web accessible on the Internet at 
http:/Avww.intel.com/quickweb; Spyglass Prism accessible 
on the Internet al http:/Avww.spyglass,com/products/prism; 
A. Fox and E. A. Brewer, "Reducing WWW Latency and 

5 Bandwidth Requirements by Real-time Distillation," In 
Proc. Of the 5th International WWW Conference, 1996; and 
T. W. Bickmore and B. N. Schiht, "Digesior: Device- 
independent Access to the World Wide Web," In Proc. Of the 
6th International WWW Conference, 1997. The proxy 

10 distills, or transcodes, the content from the Web server. This 
transcoding is primarily limited to the compression of 
images, or a reduction of their size or color space. These 
systems do not consider transcoding into different modali- 
ties. The image compression and size reduction pohcies are 

15 static and do not dynamically account for resources on the 
client. 

SUMMARY OF THE INVENTION 

The present invention adapts multimedia content, e.g., 
Web documents, to optimally match the capabilities of the 
client device requesting it. Each Web document is a set of 
items, each of which is authored in a particular modality 
such as text or image. Each of these content items is then 
transcoded into multiple resolution and modality versions so 

^ that they can be rendered on different devices. For example, 
a video item is transcoded into a selected set of images so 
that it can be rendered on a device not capable of displaying 
video. Each version of a content item requires different 
resources from the client device. The invention ensures that 
the resource requirements for the entire document, as given 
by the sum of the resource requirements of its constituent 
items, can be met by the requesting chent. The invention 
allocates the resources on the client among the items in the 
document. This resource allocation results in the selection of 
appropriate resolution or modality of the content items. If 
the client has limited resources, e.g., such as a PDA or pager, 
some of the content items may not get any resources 
assigned and thus not be delivered to the client. 

In an embodiment of the invention, as will be explained, 
three technologies are employed to provide such multimedia 
content adaptation: (i) a progressive data representation 
scheme referred to as the InfoPyramid as described in C-S. 
U, R. Mohan and J. R. Smith, "Multimedia Content 

45 Description in the InfoPyramid," Proc. ICASP'98, Special 
Session on Signal Processing in Modern Multimedia 
Standards, Seattle, Wash., May 1998, the disclosure of 
which is incorporated herein by reference; (ii) a set of 
transcoding modules for convening modality or resolution; 

jQ and (iii) an adaptation process that selects the best repre- 
sentation to meet the client capabilities while delivering the 
most value to the client. 

The present invention provides many advantages over 
prior solutions. For example, content is dynamically adapted 

55 to the client device allowing a wider variety of multimedia 
content and of client devices to be properly supported. Also, 
in accordance with the invention, a content author has 
control over the adaptation process. The content author can 
edit and replace the transcoded versions of content items 

50 generated by the automated transcoding systems. This con- 
trol of the customization overcomes problems of pubUsher 
control and copyright issues faced by transcoding proxies. 

Further, the invention permits content to be authored in 
XML (Extensible Markup Language, as is known in the art), 

65 allowing the author to provide more information to the 
transcodmg and adaptation systems than can be deduced 
from an HTML (hyper text markup language) page. One 
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benefit of the server-based system of the invention is that 
due to the guidance provided by the author, a significantly 
greater level of customization can be performed than is 
possible in previous transcoding proxies. Still further, the 
invention permits the transooded versions of the content 
items to be generated prior to any requests. Thus, the 
invention can handle media items such as video and audio 
which are diflScult to handle in conventional proxies. This 
off-line transcoding also leads to lower response latencies 
than proxies. Also, the server shares the benefit of transcod- 
ing proxies in speeding content delivery as the customized 
content is often much smaller than the original content. 

These and other objects, features and advantages of the 
present invention will become apparent from the following 
detailed description of illustrative embodiments thereof, 
which is to be read in connection with the accompanying 
drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a diagram illustrating an example of a multi- 
media document; 

RG. 2 is a diagram illustrating transcoding a content item 
into an InfoPyramid; 

FIG. 3 is a flow chart illustrating a method of adapting a 
document to a client device according to an exemplary 
embodiment of the invention; 

FIG, 4 is a flow chart illustrating an adaptation process 
according to an exemplary embodiment of the invention; 

FIG. 5 is a flow chart illustrating a method of obtaining an 
adaptation process according to an exemplary embodiment 
of the invention; 

FIG. 6 is a flow chart illustrating a method of selecting 
versions of content items for a customized document accord- 
ing to an exemplary embodiment of the invention; 

RG. 7 is a block diagram illustrating deployment of an 
adaptation process in the Internet according to an exemplary 
embodiment of the invention; 

FIG. 8 is a block diagram illustrating an exemplary 
hardware architecture of a server or computing device for 
implementing the invention; and 

FIG. 9 is a graphical representation iUustrating a resource 
aUocation technique according to an exemplary embodiment 
of the invention. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

It is to be appreciated that the term "processor** as used 
herein is intended to include any processing device, such as, 
for example, one that includes a CPU (central processing 
unit). The term "memory" as used herein is intended to 
include memory associated with a processor or CPU, such 
as, for example, RAM, ROM, a fixed memory device (e.g., 
hard drive), a removable memory device (e.g., diskette), etc. 
In addition, the term "input/output devices" or "I/O devices" 
as used herein is intended to include, for example, one or 
more input devices, e.g., keyboard, for making requests for 
multimedia content and inputting data to the processing imit, 
and/or one or more output devices, e.g., CRT display and/or 
printer, for presenting requested multimedia content and 
providing results associated with the processing imit. It is 
also to be understood that various elements associated with 
a processor may be shared by other processors. Accordingly, 
software components including instructions or code for 
performing the methodologies of the invention, as described 
herein, may be stored in one or more of the associated 
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memory devices (e.g., ROM, fixed or removable memory) 
and, when ready to be utilized, loaded in part or in whole 
(e.g., into RAM) and executed by a CPU. 

Referring to FIG. 1, an example of a multimedia docu- 

5 ment is shown. As is well known, a Web user (client) makes 
a request from his client device for some particular Web 
document(s) or other multimedia content which may contain 
video, images, graphics, text, audio. The request is sent to a 
Web server which satisfies the request by returning the 

10 requested document to the client. As shown, a multimedia 
document W, denoted as reference numeral 100, is com- 
posed of a number of content items A,-, denoted as reference 
numeral 120, where i«l, 2, . . . , n. Thus, the mxiltimedia 
document may be represented as W={A,}. Each content item 

1^ A,-, can be an atomic unit of media such as, for example, an 
image. A content item A,- can also be composed of other 
items. For example, a document can have a ninnber of 
stories as content items, and each story item may be com- 
posed of image items, text items, etc. 

Referring to FIG. 2, a diagram illustrating an example of 
transcoding a content item into an InfoPyramid according to 
the invention is shown. Particularly, a content item 120 is 
transcoded, in process 250, into an InfoPyramid 280. For 
each original content item A,-, an InfoPyramid represented as 
M,-*{M,^.}j-0, m,-, is computed by transcoding A,, into j 
versions with different resolutions and modalities. The origi- 
nal version is denoted as M,o-A,. A null version, which 
corresponds to an item being deleted from the delivered 
content, is denoted as: 

30 

CD 

The transcoding process 250 can convert the input modal- 
ity associated with the content item to a new modality such 

35 as, for example, text to audio, or audio to text. Alternatively, 
the transcoding process can summarize, compress, or elabo- 
rate on the content within the given modality of the input 
data such as, for example, by image compression or text 
summarization. Further, it is to be appreciated that iranscod- 

40 ing processes may alternatively be performed in accordance 
with the techniques described in U.S. patent application Ser. 
No. 09/413,515, entitled: "Method and Apparatus For 
Transcoding Multimedia Using Content Analysis," filed in 
the names of Li et al. on Oct. 6, 1999, which claims priority 

45 to U.S. provisional patent application Ser. No. 60/103,303, 
entitled: "Method and Apparatus For Transcoding Multime- 
dia Using Content Analysis," filed in the names of Li et al. 
on Oct. 6, 1998, the disclosure of which is incorporated 
herein by reference. 

50 The InfoPyramid 280 can be viewed as a generalization of 
multi-resolution representations, e.g., Gaussian image 
pyramids, to multimedia. The InfoPyramid provides a multi- 
modal, multi-resolution representation hierarchy for 
multimedia, for example, as described in the above- 

55 referenced C-S. Li, R. Mohan and J. R. Smith, "Multimedia 
Content Description in the InfoPyramid," Proc. ICASP'98, 
Special Session on Signal Processing in Modem Multimedia 
Standards, SeatUe, Wash., May 1998. The InfoPyramid 280 
is a data structure in which the multiple representations of a 

60 muhimedia item can be organized into a pyramid -like 
structure, as illustrated in FIG. 2. The cells of the pyramid 
correspond to different representations of the objects using 
different modalities such as, for example, video 281, image 
282, audio 283 and text 284, and fidelities such as in the 

65 range of full-resolution (bottom of pyramid) to low- 
resolution (top of pyramid). A specific version of a multi- 
media item can be referred to by one of the cells. The 
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InfoPyramid may include procedures and rules for translal- transcoded (250) into multiple modality and fidelity versions 

ing and summarizing (transcoding) between modalities and to generate a set 340 of InfoPyramids 280. 

resolutions. The InfoPyramid may also contain meta-data A client device is characterized by its profile 310 whicb 

for each constituent version such as, for example, size, color, lists the capabilities 320 and resources 360 of the device, 

bandwidth requirements, publisher preferences, etc. s The types of client devices that can access the Internet are 

The transcoding process 250 populates the InfoPyramid rapidly expanding beyond the typical workstation on a LAN 

280 structure with multi-resolution, multi-modal versions of "^^^^^ ™st multimedia Internet content is authored, 

the content item, as explained herein. The transcoding A client may have a different number of capabilities 320 

process 250 can be performed on a version by foUowing the 'f^^'^^ ^ ^ appreciated that the term 

transcoding paths in FIG. 2, for example, as denoted by 10 "c^P^^l^^ty is the abihty for a chent device to handle a 

£ 1 .u u ^t\i\ n r 11 * *u particular media typc. Examples of capabihties that may bc 
reference numerals 285 through 290. By foUowine the ^ *; j • i ^ A i /• •^ j- u v. ■^ 
, . t. ^ • u specified include: (i) screen color; (ii) audio capability; (ui) 
honzont^ paths, e.g., paths 285 and 286, a version can be capability; (iv) image display capabiUty; (v) software 
translated to another version in a new modality. For ^^^^ ^ ^^eo decompression; and (vi) hardware such as a 
example, text can be converted to audio m path 286. sound card, to name a few. So, by specifying the capabUities 
Similarly, video can be converted to images m path 285. By 15 that a client device has, it is implied that capabilities not 
following the vertical paths, e.g., paths 287, 288, 289 and specified do not exist on the client device. For example, a 
290, a version can be changed into versions at different handheld personal computer (HPC) may not be capable of 
fidelities. For example, text passages can be summarized in displaying video and a PDA may not be capable of display- 
path 288, video can be compressed in path 287 and/or ing color images. The "resources" of a client can be divided 
images can be compressed in path 289. Furthermore, the 20 up among several items on a Web page. Examples of 
image can be compressed to a greater degree in path 290. resources are: (i) screen size including width, height and 
Such transcoding processes can be done in a "lazy" fashion, area; (ii) bandwidth, for example, network bandwidth; (iii) 
i.e., the acmal transcoding need not be performed, but rather, memory and other client storage capacity; (iv) wait time 
only a place holder for the transcoded version can be created (including preferred and limits) as determined by the prod- 
and the meta-data filled in. The actual version is generated 25 uct of the bandwidth and the time a client is ready, willing 
only if required. or able to wait to receive the complete document; (v) 

Some exemplary transcoding processes 250 are listed payment ability (including preferred and limits), that is, 

below by modaUly of the version and the conversions money client is ready, willing or able to pay for the 

performed to generate versions at different resolutions and document, to name a few. 

modalities: 30 The client device profile can be obtained by a number of 

(i) images mechanisms. For example, the User- Agent fields in the 
resolution — spatial size reduction, color depth reduction, HTTP (hyper text transport protocol) request header contain 

lossy/lossless compression information about the browser and often the operating 

modality^elated text, embedded text, semantic labels; system. Also, Wiodows-CE devices specify the screen size, 

(ii) video color depth and processor. Standardization efforts are under 
resolution-spatial and temporal resolution, lossy/ way to allow these request fields to contain more informa- 

lossless compression ^ ^ ^^^^ device. Further, many sites requu-e users 

... . J. . to log- in or place cookies at the user location allowing client 

modahty-key-frame images, audio track, closed caption; capabilities to be retrieved from stored profiles. The cUent 

{m) text jj^^y specify their capabilities explicitly, for example, 

resolution— key-terms, text summarization, document rhrough forms or applets, 

headings ^ content adaptation process 350 uses the client profile 

modality-^speech synthesis, language translation; 310 to select from among the InfoPyramids 280 the versions 

(iv) audio 374 that best satisfy the particular client profile. These 

resolution— bit-rate reduction, sampling rate change, ste- 45 selected versions are rendered into a document 370 which is 

reo to mono an adaptation (i.e., customization) of the original multimedia 

modality — speech recognition, document 100. The client device receives the customized 

That is, such known transcoding operations are capable of document 370. 

transcoding multimedia content in a variety of ways. Some Referring to FIG. 4, a flow chart illustrating an adaptation 

illustrative examples of the exemplary conversions men- 50 process 350 according to the invention is shown. First, in 

tiooed above include: (i) compressing data with or without step 420, the versions from the InfoPyramids 280 that do not 

information loss; (ii) reducing the size or resolution of the meet client capabilities 320 are filtered out. For example, all 

data such as by spatial reduction; (iii) reducing the color versions that are in video modality can be filtered out if the 

content of the data such as reducing the number of colors; device is not capable of displaying video. Similarly, all 

(iv) summarizing content such as text passages and video 55 versions that are color images can be filtered out for a device 

sequences; (v) reducing or expanding the amount of data or that has only a black and white or gray-level screen, 

altering the fidelity of the information; (vi) translating the Next, in step 450, the client resources are allocated 

data to different modalities such as text to speech, video to between the InfoPyramids 280 and the content items. The 

images, and speech to text; (vii) translating text to a different resources can be allocated by using any allocation resource 

language such as English to Japanese; and (viii) removing, 60 policy. For example, the resources can be distributed in 

substituting, blocking, emphasizing or de-emphasizing indi- proportion to the resources used by the original versions. A 

vidual multimedia objects, modal elements, or other content, method for allocating resources may be based on 

Referring to FIG, 3, a flow chart of a method 300 of optimization, as will be explained in the context of FIG. 5. 

adapting a multimedia document 100 to a client device Then in step 470, the resources allocated for each InfoPy- 

according to the invention is shown. A client device is 65 ramid are used to select the appropriate version 374, which 

characterized by a client profile 310. As shown in FIG. 2, the has not been previously fihered out in step 420, to render the 

content items 120 of a multimedia document 100 are customized multimedia document 370, 
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Referring to FIG. 5, a flow chart iliustrating a method 450 
of allocating client resources using optimization according niax fv^ 1 (5) 

to the invention is shown. First, in step 520, each content / X*^' ^ ''*^*"»' 

item is assigned a priority. The author of the Web docximeot 
may have a mental priority ordering of the items in the 5 

document. The priorities may be assigned by the author of where V,€{V(Mjy)} and R(€{R (M^)} are the values and the 
the page. Many Internet applications, such as search resources used by the i** item M, of the multimedia docu- 
engines, customized news sites, etc., generate documents ment. While V,. and R,- are discrete, we will first consider 
dynamically in response to a user request. In these item to be continuous, and then deal with the discrete case, 
applications, there is often a priority implicitly assigned to lo Rc/«:m is the maximum resource available at the client, 
the items. For example, in image search engines, the match the value V,- be some function of the resource R,-, i.e., 

scores of the returned images serve as priorities. Priorities ^rWRiO- We convert the above constrained optimization 
can be assigned based on match scores for various dynami- P^^^J^"^ (equation (5)) to an unconstrained optimization 
cally generated pages. For classes of multimedia documents, P^^'^^^"^ considering the Lagran^ opUmization tech- 
priorities can be assigned on attributes of content items. For 15 described m R^Fletcher, Practical Methods of 
^ , u • L- u • u J- optimization," John Wiley & Sons, 1987. Using the 
example. tiUes can be given a higher pnonty than headings J ...hnj tion (5) may be expressed as: 
which can be given a higher pnonty than paragraphs. \ / j r 
Content analysis or heuristics give hints to semantics or 

purpose of various content items. Such content analysis, for i(/f, A) = | V + MRcUt„, - V ]| with A i o. 

example, is described in S Paek and J. R. Smith, "Detecting 20 \i ' \ *^ ' i jj 

Image Purpose in World Wide Web Documents," In Symp. 

On Electronic Imaging: Science and Technology — m. -en - ^ ^ . /i-x 

*n c I i r T ior»o .u Then, if Rq IS au Optimal solution to CQuation (5), thcrc cxists 

Document Recognition, San Jose Cabf., January 1998, the ^ ^ VL(R.X)=0. Thus, thrcandidate kutions for 

disclosure of which « incorporated herem by reference, maximum are given by: 

while heuristics are, for example, descnl>ed in J. R. Smith, 25 

R. Mohan and C-S. Li, "Content-based Transcoding of at dL dL 6L il) 

Images in the Internet," In IEEE Inter. Conf. On Image aR~ ^ aT^" Jff""'Jr^°- 

Processing {lClP-98), Chicago, lU., October 1998, the dis- ' ' i oa 

closure of which is incorporated herein by reference. For 

example, one can categorize images as pictures, graphics, 30 Given that the items and theirvalues are independent of each 
advertisements, etc. These categories can be used to assign other, we get: 
priorities. Also, priority may be assigned to items based on 

subjective or objective importance. In any case, let P,. be the £^ _ dfi{Ri) _^ (8) 

priority assigned to item i. . 5/?, " df?, 

Next, in step 540, each version of a content item in an 35 

InfoPyramid is assigned a value. A value is a subjective tt. r ,u j j * i *• . *■ /^x 

i-^ji-. . . Therefore, the candidate solutions to equation (5) are given 

measure of fidehty and may be represented as: ^^y. 

perceived vahie of transcoded version (2) ^/rp^ AftR^ ^ftit^ ^o^ 

perceived value of ongmal Mjo 40 — r- — = — — — . „ ••• ~ A. 

aH\ aK2 aKi 



V e [0, 1], (3) 

(1 for origiiial iicra Mio 

0 when the item is excluded from Mi^j 

V provides a measure of fidelity that is applicable to 
iranscodings of media at multiple resolutions and multiple 
modalities. This also allows comparison of content items 
that are in different media types. The value V can either be 
assigned by the author for each transcoding, or a functional 
relation between V and R (i.e., the resource utilized) can be 
defined. In the special case where a distortion D of all the 
versions can be measured, and the distortion for the null 
version is assumed to be infinite, V can be computed as: 

1 (4) 

V = . 

1+D 

The prioritized value of InfoPyramid is defined as 
Vf^»P,-V,.. Next, in step 560, the client resources 320 are 
allocated by maximizing, or at least substantially 
maximizing, the sum of prioritized values of the InfoPyra- 
mids of all content items such that the client resources are 
not exceeded, i.e.: 



First we consider analytic functions. Content value, as an 
alternative distortion, makes it possible for authors or users 
45 to specify value judgements about various transcoded ver- 
sions of the content. However, manually assigning the 
values is not a practical proposition in most scenarios. To 
mitigate this problem, we introduce functional mappings 
between content value and resource utilization. 

Let us assume a function V^-=/X^i)* Note that and 
therefore, the solution, is dependent on the choice of units 
for R, If f is concave, equation (9) gives us the optimal 
solution. We will first however consider the case when f is 
not concave followed by the case when it is concave. 

Non-concave case: We will limit the discussion to the case 
when f is either linear or convex. Let us assume that the 
value of an item is linearly proportional to the resource that 
it utilizes, i.e., V,-=c,-R,-. From the definition of V we have that 
//R^=0 when item i is absent from the delivered document, 
i.e., R,=0 and fj(R^)^l for the original version of item i, i.e., 
R.oR,.'~«. Thus: 

1 (10) 

ct 

65 

We term c^ to be the resource utiUzation factor RUF because 
it measures how well the item i utilizes its resources to 
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deliver value. It is evident thai a greedy algorithm that 

allocates resources to items in the order of their RUFs gives y ^. (14) 

the optimal resource allocation: , 

1. Store items in order of decreasing RUF, c, 

2. Starting with the item with the largest RUF, aUocatc the ^ ^ ^^^^ ^^^^ ^ ^^^^ ^^^^-^^ ^^^^^^ 
maximum resources that each Item can use until all the ^ ^j^^^^ ^ 

resources are depleted. _ • * r • j . t u * u i 

. , „ - ^ transcript or video may take more screen space but nave less 

Similarly the optimal r^ource aUocation for any convex ^^^^ so it is out of the solution space, 
function f IS also a greedy algonthm. ,i , ,^ . . . - • i • • . . 

^ . .f , i^.,, Recall that P.- is the priority assigned to item i by the 

Concave case: Ut us conader he concave flinclton V.^c, ^^^^^ ^ appUcalion. We then define the prioritized 
b (l+RD- We have defined f on UR, to avoid negative V, j ^ V.^=P,V,. Tlie goal is now to find; 

For simphcity, we assume that R»l for most versions, and i i » » 

that R=l is equivalent to the item being deleted, giving: 



w l°(/?i) (11) 



T ^'"1 ^^Yj^i^ ^cticm^ 



(15) 



Using this formulation, the following resource allocation 
We now get a resource utilization factor RUF of: strategies are similar to those described above but with 

prioritized values replacing RUFs: 

^ 1 (12) 1. If / is linear or convex, the resouix:es are assigned in a 

^('^) greedy manner in order of the prioritized values of the 

items. 

Using equation (9), we see that the resources are distributed 25 2. If / is logarithmic, the resources are assigned in 
among the items in proportion to their RUFs. Since, V«c, In proportion to the prioritized values of the items. In 
(R,.) is concave (and the constraint is linear) this solution is general, when f is concave, equation (15) can be solved 
optimal. In a similar manner, equation (9) yields the optimal t»y the technique described above with respect to ana- 
solution for all other concave functions. ^y^i^ functions. 

Now considering discrete values, since each item is ^0 Ji^^J. ^ »°fly'!=; '{"yj"^ ^^^'^ f**""'** Pj°' 

iranscoded into a limited number of versions, we may have C^G-. ') '^P^'"^^ "P-^, on the y-axis. The 

no version that uses exactiy the same resource as computed «:mamder of the dgonthm is as described above with 

in the optimization process described above. To account for respect to arbitrary functions. . . . , 

the discrete values, we use the following algorithm: „9'«= '"°sc«l"'g practice is to scale the size of 

35 all images by a constant factor: 0.75, 0,25, etc. One can 

1. For each item i, let R/ be the resource selected by the ^j^^ j^at in the original Web document, the larger images 
optimization process. Select version j such that R^<R,-' were more important as more resources were given to them, 
and R/-Ry is minimum, model these intrinsic properties as proportional to 

2. Order the items in order of decreasing RUFs. Starting ln(image size). Then, this ad-hoc policy of constant scaling 
from the item with the highest RUF, while there are any 40 is equivalent to allocating the client resources of screen area 
resources left, assign to each item the version with the in proportion to the prioritized values of the images, and is 
next highest value. optimal with the assumption that image value is a logarith- 

It is to be appreciated that step 2 needs to be performed only niic function of its size. 

once. Each content item i can be a composite item, i.e., a 

Next we consider arbitrary functions. When the values V^- 45 hierarchy of other content items. To account for composite 

are assigned, for example by the autiior, we may not have an i^^ms, we allocate resources using V,«/XRi) ^h^re jp is a 

analytical function /X^i) and thus wiU not be able to concave analytic function and the items i under consider- 

calculate: composite. The optimal resource R,' thus 

allocated to each composite item i is in turn used as the 

afi(R;) (13) 50 resource constraint for its constituent items. We then allocate 

an.' • this resource R/ among the children of the composite item 

i. This resource allocation is repeated imtil the items being 
considered are atomic (e.g., an individual media item such 

For this case, we adapt a technique described in Y. Shoham as an image). When we have priorities assigned to items, we 

and A. Gersho, "Efficient Bit Allocation for an Arbitrary Set 55 similarly modify the resource allocation strategy as 

of Quantizers," IEEE Trans. On Acoust., Speech, Signal described above. 

Proc, 36(9): 1445-1453, September 1998, the disclosure of For a composite item, the number of its different versions 

which is incorporated herein by reference. For each Infopy- is combinatorial in the number of its children items. Thus, it 

ramid M,. of each item i, we plot the value V^. versus the is not always practical to manually assign values to each 

resource utilized R,y of each version M^., as illustrated in 60 version of a composite item. 

FIG. 9. The optimal version M- given by sweeping a line As previously mentioned, a client may have a different 

with slope X, from the top-left to the bottom-right until it number of capabilities and resources. Before we start the 

meets the concave hull of these points. As shown by resource allocation process, we remove from consideration 

equation (9) and as known in the above-referenced Shoham all the versions of items that a client is not capable of 

article, the optimal solution is given by the same slope \ for 65 handling. Let there be r different resources Rectum we 

all the different items i. Then, a binary search for X is have to consider. Then, the resource allocation problem can 

performed such that: be stated as: 
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These selected versions 374 are rendered as the multimedia 

(16) document 370 customized for the client device. 

\ such that ^ /?J i f^^sm for all 1 i A i r. It is to be appreciated that in an alternative embodiment, 

' the fidelity measure values, priorities, and the Infopyramids 

5 may be made part of the original multimedia document. If 

We first aUocale each resource R*^,,^ separately. Let M,* be is the case, then only the selection process (HG. 6) need 

the version of item that is selected for resource k. We performed to generate the customized multimedia docu- 

select only one version of an item to be delivered. We ment 370. 

achieve this by the foUowing algorithm: Referrmg to FIG. 7, a block diagram illustrating an 

1. For each item i find the set of resources R,={R^(M,*)} lo ^'^^"'P!*^ °f deployment of a content adaptation process 300 
used by each of its versions selected for each of the ^ccordmg to the invention in a chent-server environment is 
resources shown. It is to be appreciated that the content adaptation 

. ...L process can be deployed on the server-side, the client-side, 

2, For each item i find the version M,^ that has the highest ^r in the network connecting the server and client. For 
assigned value among aU the versions and such that for example, the content adaptation process 300 may be imple- 
aU k, R (My)^R (M,*)eR,. If no values have been 15 rented in a server 701 or a computing device 704 on the 
assigned, since we define V,-/, (R,*), the value of a server-side. Further, the process 300 may be implemented in 
version may be dififerent for different resources. In that a proxy server 705 in the network. Still further, the process 
case, we choose a resource as the dominant resource, 300 may be implemented in a client device 703 or a 
and use the values of the versions for that resource. computing device 706 on the client-side. 

The above algorithm is guaranteed to select one version 20 If the content adaptation process 300 is deployed at the 

for each item without breaking the constraints of any of the proxy server 705, the proxy handles the requests from the 

r resources. However, the version so selected may not client devices 703 for multimedia content (e.g., documents), 

always be optimal. To find the optimal set, a search (e.g., The proxy retrieves the requested information firom server(s) 

coml5inatorial) may be required. 701. Then, the process 300 running on the proxy adapts the 

For finding the optimal content adaptation schemes we 25 retrieved content to the profile 310 of the requesting client 

assumed that the content items on a Web page are indepen- device 703. The proxy 705 then delivers the customized 

dent of each other. This assumption may not hold in all document 370 to the client device 703. If the content 

cases. For example, for a news story, if the text to the story adaptation process 300 is deployed at the server 701, the 

has to be discarded due to space limitations, then delivering process 300 has direct access to the content at the server 701 

the pictures for the story may not be very useful. One 30 and can store the InfoPyramids 280 on the server 701. If the 

solution is to use composite items, as described above. We content adaptation process 300 is deployed at the client-side 

consider dependent items as composite items and allocate computing device 706, the process 300 can adapt documents 

resources first to the composite item. This resource can then retrieved by the clients 703 from the server(s) 701. 

be allocated among the components of the composite item in Alternatively, the process 300 may be distributed among two 

an all-or-none manner. Another solution is to extend rate- 35 or more locations. 

distortion techniques for handling dependent blocks such as, Referring to FIG. 8, a block diagram is shown of an 

for example, is described in A. Ortega and K. Ramchandran, exemplary hardware implementation for any of the com- 

"Rate-Distortion Methods for Image and Video puter systems used to implement the methodologies of the 

Compression," IEEE Signal Processing Magazine, Novem- invention, e.g., server(s) 701, client devices 703, computing 

ber 1998, to the value -resource framework. 40 device 704, proxy server 705, and computing device 706, 

Advantageously, given that client resource allocation is Each computer system includes a processor 810 coupled to 

expressed in the form of the above optimization problem, a memory 820 and I/O device(s) 830. The processor 810 

any of the above described techniques may be employed. It performs the functions associated with the methodologies 

is to be appreciated that the above techniques are also described herein. The memory 820 is used by the processor 

described in R. Mohan, J. R. Smith and C-S. Li, "Adapting 45 for performing such functions and for storing results of the 

Multimedia Internet Content for Universal access," IEEE processes. The I/O device(s) may include one or more data 

Transactions on Multimedia, Vol. 1, No. 1, March 1999, the input devices (e.g., keyboard, etc.) for inputting data (e.g., 

disclosure of which is incorporated herein by reference. It is requests for multimedia content), and/or one or more data 

further to be appreciated that techniques known to those output devices (e.g., display) for presenting results associ- 

ordinarily skilled in the art for computing the solution to this 50 ated with the functions performed in accordance with the 

type of optimization problem may alternately be employed. methodologies described herein (e.g., displaying customized 

Whichever technique is used to find the optimum value, the documents). Also, it is to be understood that more than one 

solution gives the amount of resource R,- allocated to the processor may be employed within any one computer sys- 

InfoPyramid M,. for content item A,-. Then, the versions are tem and various processors may ^are responsibility for 

selected based on the resources that have been allocated 55 executing processes associated with the invention. While the 

(step 470). above embodiments illustrate a processor-based implemen- 

Referring to FIG. 6, a flowchart illustrating a method 470 tation of the invention, one of ordinary skill in the art will 

of selecting versions based on the resources allocated (step realize that other implementations may be used in whole or 

450) according to the invention is shown. For each InfoPy- in part, e.g., application specific integrated circuits, special 

ramid M, for content item A,-, let R,' be the resource allocated 60 purpose circuitry, etc. 

by step 450. In step 640, for each Info Pyramid, select Although illustrative embodiments of the present inven- 

version j such that Ry =R/ and Rj'-Riy is a minimum. In step tion have been described herein with reference to the accom- 

680, the InfoPyramids are sorted in the order of decreasing panying drawings, it is to be understood that the invention 

prioritized values. Starting from the item with the highest is not limited to those precise embodiments, and that various 

prioritized value, while there are any resources left, select 65 other changes and modifications may be affected therein by 

from the InfoPyramid of each item the version with the next one skilled in the art without departing from the scope or 

highest value. Step 680 needs to be performed only once. spirit of the invention. 
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What is claimed is: 

1. A method of adapting multimedia content to a client 
device, the multimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 5 

transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabilities; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the muUimedia content; and 

selecting one or more of the transcoded versions of the 15 
multimedia content to generate a customized content 
based on allocation of the client device resources. 

2. The method of claim 1, wherein the midtimedia content 
includes at least one World Wide Web document. 

3. The method of claim 1, wherein the multimedia content 20 
includes at least one of a video item, an image item, a 
graphics item, a text item, and an audio item. 

4. The method of claim 1, wherein the transcoding step 
further includes the steps of: 

separating the muUimedia content into individual content 25 
items; and 

converting the individual content items to different reso- 
lutions and modaUties. 

5. The method of claim 1, wherein the capabilities asso- 
ciated with the client device include at least one of a color 30 
capability, an audio capability, a video capability, an image 
capability, and a text capability. 

6. The method of claim 1, wherein the capabilities asso- 
ciated with the client device includes a decompression 
capability. 35 

7. The method of claim 1, wherein the resources associ- 
ated with the client device include at least one of a screen 
size, a bandwidth, a storage capacity, a content wait time 
abiUty, and a payment abihty. 

8. The method of claim 1, wherein the transcoding step 40 
includes separating the multimedia content into individual 
content items. 

9. The method of claim 8, wherein the allocating step 
further includes assigning a priority to a content item. 

10. The method of claim 9, wherein the priority is 45 
assigned by an author of the multimedia content, 

U. The method of claim 9, wherein the priority is 
assigned by an application associated with the multimedia 
content. 

12. The method of claim 9, wherein the priority is 50 
assigned in response to a user request. 

13. The method of claim 9, wherein the priority is 
assigned in response to a match score. 

14. The method of claim 9, wherein the priority is 
assigned as a function of the importance attributed to a 55 
content item. 

15. The method of claim 9, wherein the allocating step 
further includes assigning a fidelity measure value to each 
content version. 

16. The method of claim 15, wherein the allocating step 60 
further includes computing a product of the priority and the 
fidelity value for each content version. 

17. The method of claim 15, further including the step of 
including the assigned fidelity measure value in the multi- 
media content. 65 

18. The method of claim 9, further including the step of 
including the assigned priority in the multimedia content. 



19. The method of claim 1, wherein the transcoded 
versions selected satisfy at least a portion of the resource 
allocations. 

20. The method of claim 1, further including the step of 
organizing at least one of the one or more items and at least 
a portion of the transcoded content versions associated 
therewith in a data structure. 

21. The method of claim 20, further including the step of 
including the data structure in the multimedia content. 

22. The method of claim 1, wherein the resources allo- 
cated to at least a portion of the transcoded versions of the 
content items are a function of priorities. 

23. A method of adapting multimedia content to a client 
device, the multimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 

transcoding the multimedia content into a plurality of 
transcoded content versions, the pluraUty of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabihtics; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources; 

wherein the transcoding step includes separating the mul- 
timedia content into individual content items, and 
wherein the allocating step further includes assigning a 
priority to a content item, assigning a fidelity measure 
value to each content version, computing a product of 
the priority and the fidelity value for each content 
version, and at least substantially maximizing a sum of 
the products for the content versions such that the 
resources associated with the cUent device are not 
exceeded. 

24. The method of claim 23, wherein the maximizing step 
results in determining the portion of the resources associated 
with the cUent device which are allocated to each content 
item. 

25. A method of adapting multimedia content to a client 
device, the muUimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the method comprising the steps of: 

transcoding the multimedia content into a pluraUty of 
transcoded content versions, the pluraUty of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabiUties; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on aUocation of the cUent device resources; 

wherein the resources aUocated to at least a portion of the 
transcoded versions of the content items are a Unear 
scaling function of the resources that would otherwise 
be allocated to the items. 

26. Apparatus for adaptmg multimedia content to a cUent 
device, tUe multimedia content including one or more items 
and the client device having capabilities and resources 
associated therewith, the apparatus comprising: 
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at least one processor operable to iranscode the multime- 
dia content into a plurality of transooded content 
versions, the plurality of transcoded content versions 
having different modalities and resolutions associated 
therewith, to filter out the transcoded content versions 5 
that are not compatible with client device capabilities, 
to allocate at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content, and to select one or more of the 
transcoded versions of the multimedia content to gen- lO 
erate a customized content based on allocation of the 
client device resources; and 

memory, coupled to the at least one processor, for storing 
at least a portion of results associated with one or more 
of the operations performed in accordance with the at 
least one processor. 

27. The apparatus of claim 26, wherein the multimedia 
content includes at least one World Wide Web document. 

28. The apparatus of claim 26, wherein the multimedia 
content includes at least one of a video item, an image item, 20 
a graphics item, a text item, and an audio item. 

29. The apparatus of claim 26, wherein the capabilities 
associated with the client device include at least one of a 
color capability, an audio capability, a video capability, an 
image capability, and a text capability. 



30. The apparatus of claim 26, wherein the resources 
associated with the client device include at least one of a 
screen size, a bandwidth, a storage capacity, a content wait 
time ability, and a payment ability. 

31. An article of manufacture for adapting multimedia 
content to a client device, the multimedia content including 
one or more items and the client device having capabilities 
and resources associated therewith, comprising a machine 
readable medium containing one or more programs which 
when executed implement the steps of: 

transcoding the multimedia content into a plurality of 
transcoded content versions, the plurality of transcoded 
content versions having different modalities and reso- 
lutions associated therewith; 

filtering out the transcoded content versions that are not 
compatible with client device capabihties; 

allocating at least a portion of the resources associated 
with the client device among the one or more items of 
the multimedia content; and 

selecting one or more of the transcoded versions of the 
multimedia content to generate a customized content 
based on allocation of the client device resources. 
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